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ABSTRACT 


Seven  sets  of  parallel  R&D  projects  involving  15  labora- 
tories are  examined.   Deta  gathered  by  Solution  Development  Re- 
cords— a  form  which  provides  a  weekly  estimate  of  the  probability 
of  adoption  of  the  approaches  under  consideration  as  possible 
solutions  to  a  technical  problem — and  post-project  interviews  with 
the  engineers  responsible  for  each  problem.   The  sources  of  tech- 
nical approaches,  indicated  on  the  Solution  Development  Records, 
were  obtained  during  the  post-project  interviews.   Eight  pos- 
sible sources  of  ideas  are  considered. 

Better  performing  groups  are  found  to  rely  more  than  poorer 
performers  upon  information  sources  within  the  laboratory,  for  gener- 
ation of  ideas.   A  serious  misalignment  appears  to  exist  between  the 
quality  of  ideas  generated  through  the  eight  channels  and  the  fre- 
quency with  which  these  channels  are  used  by  engineers.   Vendors 
and  analysis  and  experimentation,  by  the  engineer  himself,  appear 
to  be  over-utilized  relative  to  their  effectiveness  as  idea  sources, 
while  information  available  from  other  company  research  programs 
and  from  the  lab's  technical  staff  is  under-utilized. 


The  topic  of  this  conference  quite  naturally  directs  one's  attention 
to  the  process  of  information  transfer  -  the  generation,  storage,  sum- 
marization and  retrieval  of  the  ideas  and  data  of  science  and  techno- 
logy.  Information  is  both  the  raw  material  and  the  principal  product 
of  research  and  development,  and  as  raw  material,  technical  information 
is  certainly  one  of  the  key  determinants  of  research  program  effective- 
ness.  As  a  result,  a  substantial  effort  has,  in  recent  years,  gone  into 
the  development  of  improved  methods  and  systems  to  provide  the  researcher 
with  the  information  he  requires.   Progress  in  this  direction  is  depen- 
dent, however,  upon  our  knowledge  of  the  type  and  form  of  the  informa- 
tion needed;  the  exact  functions  for  which  it  is  to  be  used;  the  mech- 
anisms now  available  for  performing  these  functions;  and  the  inadequacies 
in  the  existing  communication  systems'  performance  for  each  function 
(Menzel,  1962). 

The  present  paper  considers  the  performance  of  the  existing  techni- 
cal communication  system  in  meeting  one  of  its  principal  functions:  the 
generation  of  technical  alternatives  for  consideration  as  potential  solu- 


tions  to  R  &  D  problems;  it  is  based  upon  data  gathered  in  a  study  of 
parallel  R&D  projects  which  has  been  underway  over  the  last  two  years. 


To  assist  in  obtaining  a  performance  evaluation  for  specific  informa- 
tion channels  instances  were  sought  out  in  which  the  same  problem  is  at- 
tempted by  two  or  more  research  groups.   In  this  way,  a  comparison  can  be 
made  between  the  information  sources  leading  to  specific  solutions.  The 
fairly  common  practice  of  the  Department  of  Defense  and  the  N.A.S.A.  to 
award  parallel  study  contracts  during  the  early  phases  of  R  &  D  provides 
an  opportunity  for  control  over  the  effect  of  variations  in  the  problem 
substance.  The  data  presented  are  from  seven  parallel  contracts  involving 
15  R  i  D  groups. 

METHODS 

Once  a  parallel  project  has  been  located,  its  work  statement  is  ob- 
tained and  analyzed  and  factored  into  a  reasonable  number  of  sobproblem 
areas  (generally  subsystems).   The  breakdown  is  then  checked  with  the 
technical  person  who  prepared  the  work  statement,  and  data  collection  forms 
based  upon  it  are  designed.   After  all  data  have  been  collected  from  the 
contractors,  the  technical  monitor  is  revisited  and  asked  to  provide  a  con- 


I 

The  idea  of  studying  parallel  R  4  D  projects  as  a  quasi-experimental 
situation  was  suggested  by  Donald  Marquis,  who  also  helped  in  developing 
some  of  the  research  procedures. 
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fidential  evaluation  of  each  lab's  performance  on  each  subproblem.   Data  are 

gathered  by  four  means:   (I)  time  allocation  forms,  indicating  the  amount 
of  time  each  engineer  spends  on  the  job  in  several  activity  categories;  (2) 
before  and  after  interviev/s  with  the  individual  engineers;  (3)  periodic  tape 
recorded  progress  reports  by  the  project  manager;  and  (4)  solution  development 
records. 

Post-project  interviews,  supported  with  the  charted  Solution  Development 
Record,  are  the  principal  source  of  the  data  here  under  consideration.   The 
Solution  D  veiopment  P  cord  is  a  research  +oo  I  which  provides  a  record  over 
time  of  the  progress  of  an  individual  engineer  or  group  of  engineers  (or 
scientists)  toward  the  solution  of  a  technical  problem.   The  lead  engineer 
responsible  for  each  subproblem  is  asked  to  provide  a  weekly  estimate,  for 
each  alternative  approach  under  consideration,  of  the  probability  that  it 
will  be  finally  chosen  as  the  solution  to  that  subproblem. 

Figure  I  illustrates  the  listing  of  alternative  approaches  identified  from 
the  contract  work  statement,  when  so  specified,  and  from  the  responsible  engineer 
when  he  is  interviewed  prior  to  beginning  the  task.   Blank  spaces  are  always 
provided  so  that  new  approaches  may  be  reported  as  they  arise.   It  at  some 
point  in  the  design  the  respondent  were  considering  two  technical  approaches 
to  rendezvous  at  Uranus,  and  he  were  completely  uncommitted  between  the  two, 
he  would  circle  0.5  for  each,  as  shown.   Eventually  as  the  solution  progresses, 
one  alternative  will  attain  a  1.0  probability  and  the  others  will  become  zero. 
By  plotting  the  probabilities  over  time,  we  obtain  a  graphic  record  of  the 
solution  history. 


FIGURE  1 

MASSACHUSETTS  INSTITl/TE  OF  TECHNOLOGY 
Solution  Development  Record 

Manned  Uranus  Landing  in  an  Early  Time  Period  Study 
General  United  Aerospace  Corporation 
me  _^_    Date , 


Estimate  of  Probabi lity  that  Alternative  w! 1 1 
I  be  Employed 

bproblem  #1:  Method  of 
ndezvous  at  Uranus 

ternotive  approaches: 

orbital  rendezvous  mission     0  0.1  0.2  0.3  0.4  (o.s)  0.6  0.7  0.8  0.9   1.0 
with  excursion  vehicle 

orbital  rendezvous  mission     0  0.1  0.2  0.3  0.4  0.5  0.6  0.7  0.8  0.9  1.0 
without  excursion  vehicle 


direct  mission  0  0.1  0.2  0.3  0.4^0^0.6  0.7  0.8  0.9  1.0 
0  O.i  0.2  0.3  0.4  0.5  0.6  0.7  0.8  0.9   1.0 


bproblom  #2:   Design  of  the 
ectrical  power  supply  subsystem 
>r  the  space  vehicle 

ternatlve  approachess 

hydrogen-oxygen  0  0.1  0.2  0.3  0,4  0.5  (0.6^  0.7  0.8  0.9   1.0 

fuel  eel  I 


KOH  fuel  cell  0  0.1  0.2  (0^  0.4  0.5  0.6  0.7  0.8  0.9  1.0 

Rankine  cycle  fast  reactor  0  0.1  0.2  0.3  0.4  0.5  0.6  0.7  0.8  0.9  1.0 

Rankine  cycle  thermal  reactor  0  0.1  0.2  0.3  0.4  0.5  0.6  0.7  0.8  0.9  1.0 

Breyton  cycle  reactor  0  (p7j)  0.2  0.3  0.4  0.5  0.6  .0.7  0.8  0.9  1.0 

_^_  0  0.1  0.2  0.3  0.4  0.5  0.6  0.7  0.8  0.9  1.0 

0  0.1  0.2  0.3  0.4  0*5  0.6  0.7  0.8  0.9  1.0 

0  0.1  0.2  0.3  0.4  0.5  0.6  0.7  0.6  0.9  1.0 


When  the  project  is  completed,  each  respondent  is  presented  with  a  time- 
plot  of  his  probability  estimates,  and  is  interviewed  at  some  length  to  deter- 
mine causes  and  effects  of  design  changes  reflected  in  this  record.   The  plot 

thus  provides  a  stimulus  to  the  man's  memory  and  assists  the  investigator  in 

2 
gathering  a  detailed  record  of  each  project. 

Description  of  the  Project" 

The  seven  projects  under  consideration,  involved  the  following  general 
problems: 

1.  The  design  of  the  reflector  portion  of  a  rather  large  and 
highly  complex  antenna  system  for  tracking  and  communication 
with  space  vehicles  at  very  great  distances. 

2.  The  design  of  a  vehicle  and  associated  instrumentation  to 
roam  the  lunar  surface  and  gather  descriptive  scientific  data. 

3.  An  investigation  of  passive  methods  for  transfer  of  modulation 
between  two  coherent  light  beams. 

4.  A  preliminary  design  of  an  earth-orbiting  space  station. 

5.  The  design  of  a  deep  space  probe,  and  appropriate  instrumenta- 
tion. 

6.  The  preliminary  design  of  an  interplanetary  space  vehicle. 

7.  The  preliminary  design  of  a  spec ia j-purpose  manned  spacecraft 
for  cislunar  missions. 


2 

Solution  Development  Records  were  not  obtained  or  were  of  poor  quality 
for  two  of  the  seven  projects.   In  these  instances,  engineers  were  asked 
to  recap,  as  best  they  could,  their  solution  process,  including  all  technical 
approaches  considered.   Without  the  benefit  of  the  stimulus  which  the  solution 
development  plot  provides  to  the  man's  memory,  some  considered  approaches  may 
well  have  been  missed.   While  this  reduces  the  size  of  the  sample,  it  should 
not  bias  it  toward  or  away  from  specific  channels. 
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Functions.  Channels  and  Messages 

While  the  technical  communication  system  performs  many  functions  for 
the  engineer,  the  focus  here  will  be  on  one:   that  of  generating  technical 
alternatives  to  the  solution  of  R  &  D  problems.   Eight  channels  from  which 
such  alternatives  can  be  obtained  are  considered  (Table  I).   In  addition 
to  the  six  channels  external  to  the  engineering  group,  two  sources  of  ideas 
(analysis  and  experimentation  and  personal  experience),  which  are  internal 
to  the  group  are  included  in  the  analysis. 

The  unit  of  analysis  employed  is  "messages  received"  (Menzel,  I960, 
pp.  20-22).   In  other  words,  the  suggestion  of  a  potential  solution  to  the 
problem  is  considered  a  message  received.   Quite  often  a  single  alternative 
will  result  from  messages  received  from  several  channels;  for  example, 
reference  in  the  trade  literature  might  lead  the  engineer  to  a  vendor  who 
provides  more  complete  information  on  the  alternative.   In  such  a  situation, 
where  several  channels  contribute  to  a  single  alternative,  credit  is  given  to 
each  source.   Sources  of  alternatives  were  sought  out  in  the  post-project  in- 
terviews, and  the  tape-recorded  transcriptions  of  these  interviews  were  coded 

3 
to  indicate  channels  through  v/h  i  ch  each  alternative  was  generated.   A  total 

of  482  technical  alternatives  resulted  from  610  messages  received;  179  of 

these  messages  were  finally  accepted  resulting  in  117  problem  solutions. 


3 
An  inter-coder  reliability  of  80^  perfect  agreement  was  obta'nned  for  a 
sample  of  the  transcriptions. 


TABLE   I 


Information  Channels  Considered  in  the  Study 


1  iterature: 


vendors: 


books,  professional,  technical  and  trade  journals, 
and  other  publicly  accessible  written  material. 

representatives  of,  or  documentation  generated  by, 
suppliers  or  potential  suppliers  of  design  com- 
ponents. 


customer: 


externa  I 
sources: 


techn  i  cal 
staff: 


representatives  of,  or  documentation  generated  by, 
the  government  agency  for  which  the  project  is 
performed . 

sources  outside  the  laboratory  which  do  not  fall 
into  any  of  the  above  three  categories.   These 
include  paid  and  unpaid  consultants  and  represen- 
tatives of  government  agencies  other  than  the  cus- 
tomer agency, 

engineers  and  scientists  in  the  laboratory  who  are 
not  assigned  directly  to  the  project  being  con- 
sidered. 


company  research:  any  other  project  performed  previously  or  simul- 
taneously in  the  lab  regardless  of  its  source  of 
fund  i  ng. 


personal 
experience: 


analysis  and 
experimentation: 


ideas  which  were  used  previously  by  the  engineer  for 
similar  problems  are  are  recalled  directly  from 
memory. 

ideas  which  are  the  result  of  an  engineering  ana- 
lysis, test  or  experiment  with  no  immediate  input 
of  information  from  any  other  source. 


RESULTS 

An  example  of  a  solution  development  plot  is  shown  in  Figure  2.   In 
this  instance,  each  group  considered  five  alternative  approaches  to  the 
problem.   Three  of  these  ( cK  ,  ^  and  Y)    were  specified  by  the  customer  at 
the  start  of  the  contract.   Ti.o  were  generated  by  each  team  during  the 
course  of  the  project.   In  team  A  approach O  had  been  previously  used  by 
the  engineer  on  problems  of  a  similar  nature,  and  the  association  of  this 
problem  with  the  former  ones  brought  the  idea  out  of  memory.   Approach  ^ 
was  suggested  by  a  paper  which  a  colleague  had  heard  presented  at  a  SAE 
meeting.   Further  details  on  this  approach  were  obtained  through  the 
trade  literature  and  vendors. 

In  team  B  an  engineer,  thumbing  through  a  colleague's  reference  file 
of  clippings  from  trade  journals,  ran  across  an  item  which  suggested  ap- 
proach/ to  him.   He  had  employed  this  approach  before  but  it  had  not  oc- 
curred to  him  as  a  possibility  for  this  particular  application  until  his 
memory  was  jolted  by  the  trade  journal  clipping.   Fortuitous  contact  with 
the  representative  of  a  vendor  firm  suggested  approach  "^  .   This  represen- 
tative happened  to  visit  a  man  in  another  department  who  knew  of  the  en- 
gineer's problem  and  suggested  that  the  vendor  contact  him.   In  a  similar 
manner,  alternatives  considered  in  other  subproblems  are  attributed  to  the 
information  channels  whence  they  originated. 

The  alternatives  are  evaluated  at  two  levels.   At  the  first  level,  the 


TABLE 


Messages  Received  and  Messages  Accepted  by  R  &  D  Engineers 
as  a  Function  of  Information  Channel 


channe I 


messages 
rece  i  ved 


messages 
accepted 


acceptance 
rat  io 


(D 
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JD 
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I i terature 
external  sources 
vendors 
customers 


45 
50 
80 
99 


14 
24 
25 
23 


0.31 
0.48 

0.31 
0.23 


tecinnical  staff 
~  o   company  research 

Ul  +- 

C  (0 

o   ana  lysis  and 

—  (0    experimentation 
0  — 

c 

ro  j:        personal    experience 

JZ    +- 


27 

23 

164 

50 


13 
13 
47 


0.48 
0.57 
0.29 

0.30 


unl<nov;n 


72 
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engineer  himself  decides  upon  one  of  several  possibilities  as  the  best  solution 
to  the  subproblem;  he  thus  evaluates  this  alternative  as  preferable  to  the 
others  under  consideration.  At  a  second  level,  for  at  least  some  of  the 
subproblems,  a  relative  evaluation  by  the  customer  of  the  solutions  reached 
by  the  two  or  three  research  groups  on  each  problem  is  available.   So,  for 
each  problem,  two  or  three  alternatives  are  first  chosen  over  all  others  by 
the  R&D  groups  and  then  these  two  or  three  are  evaluated  relative  to  each 
other  by  the  government  technical  representative.   V'orking  backward  to  the 
sources  of  the  alternatives  provides  two  measures  of  performance  for  the  in- 
formation channels. 

Acceptance  by  the  Engineer 

Table  II  shows  total  frequency  counts  for  messages  received  and  accepted 
from  each  of  the  eight  channels.   Seventy-two  alternatives  could  not  be  at- 
tributed to  a  channel,  either  through  the  unavailability  for  interview 
of  the  knowledgeable  engineer,  or  through  failure  to  obtain  this  informa- 
tion during  the  interview. 

These  data  show  quite  a  diversity  in  both  relative  use  and  perfor- 


mance of  the  eight  channels.   A  chi-square  test  performed  on  accepted  and 

4  ; 

rejected  (tota I -accepted)  messages  shows  a  significant  (X~  =  20;  p<O.OI) 

difference  in  the  allocation  of  acceptances  and  rejections  among  channels, 


4 
Alternatives  of  unknown  origin  are  excluded, 
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External  sources,  technical  staff  and  company  research  supply  a  considerably 
greater  number  of  acceptable  approaches  than  would  be  expected  by  chance, 
while  the  apprcjches  suggested  by  the  customer,  and  those  arrived  at  through 
analysis  or  experimentation  have  a  relatively  low  level  of  acceptance.   The 
distribution  of  acceptances  and  rejections  among  other  channels  is  rather 
close  to  chance  expectation. 

Interestingly,  the  acceptance  ratios  for  the  eight  chon;iels  are  inversely 
related  to  the  degree  to  which  the  channels  are  used.   Since  some  authors 
have  suggested  frequency  of  use  itself,  as  an  index  of  channel  value,  with 
the  assumption  that  a  researcher's  behavior  implicitly  recognizes  quality, 
the  finding  of  an  inverse  relation  between  frequency  of  use  and  this  first 
measure  of  channel  performance  is  surprising  and  rather  important. 

Evaluation  by  the  Customer 

For  16  of  the  63  subproblem  pairs,  relative  ev,3luations  of  the  solutions 
were  obtained  from  responsible  technical  monitors  in  the  customer  agencies. 
In  the  remaining  47  pairs,  scores  were  either  tied  or  no  evaluation  was 
available   This  relative  evaluation  permits  a  comparison  of  the  information 
channels  used  to  arrive  at  a  solution  judged  superior  to  the  solution  pre- 
sented by  other  teams. 

The  hypothesis  to  be  tested  here  is  based  upon  the  findings  of  Allen 
(1964)  for  R&D  proposal  competitions.   The  use  of  information  sources  out- 
side of  the  laboratory  was  found  to  be  inversely  related  to  the  technical 
quality  of  proposals.,  while  use  of  sources  within  the  lab  was  weakly  but 
positively  related  to  quality.  The  hypothesis  predicts  that  poorer  per- 
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TABLE  I  I  I 


Sources  of  Higher  and  Lower  Ra+ed  Solu+ians 
(  16  Subproblem  Pa!  --s) 


information  subproblems  with       subproblems  with 

channel  higher  rated  lower  rated 

solutions  solutions 


I i terature  3  4 

external  sources  2  6 

vendors  6  5 

customer  12  II 

technical  staff  2  I 

company  research  3  I 

analysis  o  experimentation  II  10 

personal  experience  3  3 

unknown  2  4 
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forming  groups  will  rely  more  heavily  upon  sources  outside  of  the  lab,  and 
better  performing  groups  more  upon  sources  within  the  lab. 

Table  III  shows  the  frequency  distributions  for  each  channel  of  mes- 
sages leading  to  higher  and  lower  rated  solutions  to  the  32  subproblems. 
Since  expected  frequencies  in  individual  cells  are  too  small  for  statistical 
test,  the  channels  were  aggregated  into  three  categories:   those  within  the 
lab,  (technical  staff,  company  research,  analysis  i  experimentation,  and 

personal  experience);  those  which  are  outside  of  the  lab  (literature,  ex- 

5 
ternal  sources,  and  vendors);  and  the  customer.   (Table  IV) 

TABLE  IV 


Sources  of  Higher  and  Lower  Rated  Solutions 
(15  Subproblem  Pairs;  Channels  Combined) 


number  of 
higher  rated 
sol ut  ions 


number  of 
lower  rated 
sol utions 


channe I s  outs  i  de 
of  the  laboratory 
(L  +  ES  +  V) 

channels   v/ithin 

the  laboratory 

(TS  +  CR  +  A/E  +  RE) 

customer 


1  I 


12 


11 


The  customer  is  treated  as  a  separate  category  since  he  is  obliged  to 
provide  information  to  both  teams  impartially. 
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TABLE  V 


Comparison  of  Sources  of  All  T'chnicat  Alternatives  Considerecj 
by  G'^oups  Submitting  Solutions  Receiving 
Higher  ancj  Lower  Ratings 
(16  Subproblem  Pairs) 


information  higher  ratecj  sub-       lower  ratecJ  sub- 

channel problem  solutions       problem  solutions 


I iterature  4  6 

external  sources  2  10 

vencjors  8  12 

customer  27  29 


technical  staff  3  I 

t  2.  company  research  4  I 


i2  °    analysis  &  experimentation        32  23 


§  Qj    personal  experience  2  3 

u  +- 


unknown 


16 


While  the  trend  is  in  the  predicted  direction,  a  chi -square  test  shows  the 

difference  in  performance  of  the  aggregate  channels  to  fall  short  of  stat I s- 

2 
tical  significance  (X  =  1.15,  p>0.50). 


Although  no  strong  conclusions  can  be  reached  concerning  the  direct 
sources  of  the  solutions  submitted  by  the  better  and  poorer  performing 
research  teams,  there  is  a  possibility  that  significant  differences  exist 
in  their  general  information-gathering  behavior. 

To  test  this,  the  sources  of  all  alternatives  considered  by  teams 
submitting  higher  and  lower  rated  solutions  are  tabulated  in  Table  V. 
Expected  frequencies  are  again  too  low  for  test,  so  channels  are  aggregated 

as  before  (Table  VI).   A  chi'^square  test  rejects  the  null  hypothesis  of 

2 
no  difference  at  the  0.05  level  of  significance  (X  =  8.71). 

Poorer  performing  teams  rely  much  more  upon  vendors,  consultants  and 
other  persons  outside  of  the  laboratory  for  their  ideas.   Better  performers 
use  their  technical  staff  to  a  greater  extent;  profit  from  information 
generated  by  other  research  programs  within  the  company;  and  generate  more 
alternatives  through  their  own  analysis  and  experimentation.   The  two  dif- 
fer little  in  their  use  of  the  literature  or  in  their  reliance  upon  their 
own  personal  experience. 

Channel  Ratings 

Since  a  numerical  score  ranging  from  I  to  5  was  assigned  each  solution 
by  the  technical  evaiuator,  this  score  could  be  assigned  to  the  channels 


TABLE  VI 


Sources  of  All  Technical  Alternatives  Considered  by  Groups 
Submitting  Higher  and  Lower  Rated  Solutions 
(16  Subproblem  Pairs;  Channels  Combined) 


subproblems  with 
higher  rated 
sol utions 


subproblems  with 
lower  rated 
solutions 


channe I s  outsi  de 
of  the  laboratory 
(L  +  ES  +  V) 


14 


28 


channels  within 
the  laboratory 
(TS  +  CR  +  A/E  +  PE) 


28 


customer 


27 


29 
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whence  each  solution  originated,  and  mean  ratings  derived  for  the  channels. 
To  control  for  differences  among  evaluators,  each  evaluator's  scores  were 
normalized  on  the  basis  of  the  rank  given  each  channel.   Ranks  were  then 

weighted  by  the  number  of  observations  on  which  they  were  based  and  mean 

6 
ranks  computed  for  each  channel  across  all  evaluators. 

The  ranking  of  information  channels,  thus  derived,  is  shown  in  Table  VII 


TABLE  VI  I 

Rank  Ordering  of  Information  Channels  on  the 
Be.sis  of  Numerical  Evaulation  of 
So! ut ions 


1 

company  research 

2 

technical  staff 

3 

personal  experience 

4 

customer 

5 

1 iterature 

6 

vendors 

7 

external  sources 

8 

analysis  &  experimentation 

DISCUSSION 
The  results  indicate  a  rather  serious  n: j sa I ignment  in  the  use  of  infor- 
mation channels  by  R  &  D  personnel.   There  is  apparently  a  substantial  var- 
iation in  the  effectiveness  of  information  channels,  as  measured  at  two 


Since  no  eva I uator  provided  scores  for  all  channels,  a  measure  of 
agreement  or  concordance  among  evaluators  was  not  derived. 
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levels,  but  this  effectiveness  is  not  recognized  and  reflected  in  the  frequency 
with  which  the  channels  are  used.  Table  VIII  illustrates  this  point  with  a 
rank  ordering  of  channels  on  the  basis  of  the  three  measures  used  in  the 
study  (message  acceptance  rate  by  the  engineer,  customer  evaluation,  and 
relative  use  of  the  channel). 


TABLE  VII  I 


Rank  Ordering  of  Information  Channels 
on  the  Basis  of  Three  Performance  Measures 


Message  acceptance  rate       Customer  evaluation       Frequency  of 
by  the  engineer  of  solutions  use 

1.  company  research  company  research-*  analysis  &  experimentation 

2.  technical  staff  technical  staff*  customer 

3.  external  sources  personal  experience  vendors 

4.  personal  experience  customer  personal  experience 

5.  literature  literature  external  sources 

6.  vendors  vendors  literature 

7.  analysis  &  experimentation  external  sources  technical  staff 

8.  customer  analysis  &  experimentation  company  research 


*t  i  ed 

Aside  from  the  placement  of  two  channels,  the  engineer  and  customer 
evaluations  are  remarkably  similar.   Even  the  two  exceptions  are  quite 
plausible  differences.   The  customer  tends  to  rate  his  own  ideas  considerably 
higher  than  does  the  engineer.   And  engineers  apparently  overrate  ideas 
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originated  by  external  sources.   I'hen  an  engineer  resorts  to  a  consultant, 
he  quite  like'y  tends  to  overestimate  the  consultant's  competence  in  the 
area  and,  as  a  result  does  not  exercise  sufficient  skepticism  in  assessing 
the  idea.   With  the  cusromer's  rating  of  his  own  ideas  disregarded,,  the  Spear- 
man rank  order  correlation  (r  )  between  the  two  levels  of  evaluation  for  the 
remaining  seven  cnannels  is  0,83  (p  -  O.OIy. 

Frequency  of  use.  however,  seems  to  run  directly  opposite  to  both  of 
these  ratings   As  mentioned  earlier,  a  comparison  of  acceptance  rate  with 
use  shows  an  almost  perfect  reverse  ordering;  in  comparing  customer  evaluation 
with  channel  usage,  r^  =  -0.67  i,p<0  05).   Preference  for  one  channel  over  an- 
other appears  to  result  from  some  feature  other  than  its  actual  usefulness. 
This  other  feature  may  well  be,  as  Ackoff  and  Halbert  (1958)  found,  the 
accessibility  of  the  channel.   Further  research  is  needed  into  the  causes  of 
the  preference  for  particular  channels. 

Comparing  channels  on  the  basis  of  their  use  and  their  contribution  to 
both  performance  measures,  we  find  analysis  and  experimentation  and  vendors 
to  be  clearly  over-ut i  i  i zed,  and  technical  staff  and  company  research  to  be 
clearly  under-ut i  :  i zed . 

It  is  difficult  to  fathom  the  meaning  behind  the  apparent  over-utiliza- 
tion of  analysis  and  experimentation   This  is  after  ail,  what  engineers 
are  paid  to  do,  and  earlier  evidence  (p.  lO;  showed  that  better  performing 
groups  used  this  source  more  than  pcorer  performers.   Perhaps  the  low  rating 
attained  here  is  a  reflection  on  the  general  quality  of  the  analysis  per- 
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formed  on  the  seven  projects.   With  vendors,  however,  there  is  a  clear  moral. 
Certainly,  good  use  can  be  made  of  information  obtained  through  vendors. 
But  in  using  vendors,  it  must  be  remembered  that  it  is  difficult  enough  for 
project  engineers  to  understand  the  customer's  needs  and  the  range  of  values  he 
is  willing  to  accept;  vendors  are  one  step  further  removed  and  the  difficulty 
increases  non- I i near ly .   Finding  a  vendor  does  not  dispense  with  the  need 
for  good  engineering  in-house.   Specifications  must  be  drawn  knowledgeab ly, 
and  vendor  activities  must  be  closely  monitored  in  order  to  assure  proper 
use  of  the  information  available  through  this  channel.  The  fact  that  vendor 
acceptance  rate  is  low  is  evidence  that  the  channel  is  used  with  some  skep- 
ticism, but  were  the  vendors  to  be  made  more  clearly  aware  of  what  was  de- 
sired of  them,  both  their  acceptance  rate  and  customer  evaluation  might  be 
improved . 

The  under-ut i I i zat ion  of  technical  staff  and  company  research  agrees 
quite  clearly  with  the  finding  that  R&D  groups  submitting  higher  rated 
solutions  rely  more  heavily,  than  do  the  poorer  performers,  upon  sources 
within  the  laboratory..   This  says  quite  a  lot  about  the  benefits  to  be 
derived  from  and  the  need  for  improving  the  communication  flow  within  the 
lab  itself.   For  example,  technical  staff  who  are  knowledgeable  in  parti- 
cular areas  could  be  made  more  readily  available  for  consulting  by  project 
members,  and  rewarded  for  the  contribution  which  they  make  through  this 
type  of  activity.   The  distribution  of  company  documents  resulting  from 
R&D  programs  and  company-sponsored  research  could  be  improved.   Perhaps 
as  the  laboratory  feels  that  it  has  attained  a  certain  degree  of  competence 


-22 


in  an  area,  summary  documentation  could  be  prepared,  presenting  the  state- 
of-the-art,  the  nature  of  the  lab's  capabilities  and  the  names  of  key  people 
who  are  available  for  consulting.   This  is  frequently  done  for  presentation 
to  customer  agencies,  but  how  often  is  such  documentation  prepared  specifi- 
cally for  in-house  use? 

Literature  does  not  appear  to  be  greatly  used,  nor  does  this  appear 
to  be  out  of  line  with  the  evaluation  of  ideas  obtained  from  it.   Perhaps 
the  most  outstanding  result  to  be  noted  here  is  that  very  few  of  the  litera- 
ture citations  are  to  the  professional  engineering  or  scientific  literature. 
Most  literature  references  are  to  trade  publications  and  free-subscription 
(advertising-supported)  technical  magazines.   A  rather  serious  problem  may 
exist  here,  since  these  publications  are  outside  the  control  of  the  profes- 
sional societies  and  are  open  to  the  temptation  of  distortion  toward  the 
products  of  their  heavy  advertisers. 

CONCLUSIONS 

This  study  has  demonstrated  the  feasibility  of  a  new  technique  for 
measuring  the  performance  of  the  technical  communication  system  and  eval- 
uating the  relative  performance  of  information  channels.   The  technique 
employs  the  vehicle  of  parallel  R&D  projects  to  provide  a  control  over 


7 
While  the  fact  that  engineers  rely  little  upon  the  literature  will 
undoubtedly  surprise  some,  it  is  a  finding  which  is  not  unique  to  this  study. 
Compare,  for  example,  Scott  and  Wilkins  (1958)  and  the  report  of  the  O.E.E.C, 
(1958). 
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the  substance  of  the  problem  and  a  relative  evaluation  of  solutions.   Data 
are  gathered  by  means  of  Solution  Development  Records  and  lengthy  interviews 
with  the  engineers.   The  ideas  considered  for  solution  to  each  problem  are 
thus  associated  with  the  channels  whence  they  came,  and  measures  of  performance 
are  generated  for  the  channels. 

The  principal  conclusions  of  the  study  are: 

1.  Better  performing  groups  rely  more  than  the  poorer  performers  upon  sources 
within  the  laboratory  (the  technical  staff,  other  company  research  pro- 
grams, and  their  own  analysis  and  experimentation)  as  contrasted  with 
sources  outside  the  lab. 

2.  There  is  a  serious  misalignment  between  the  quality  of  the  ideas  generated 
through  the  eight  channels  studied,  and  the  frequency  with  which  these 
channels  are  used  by  engineers. 

3.  Based  upon  relative  acceptance  of  messages  by  the  design  engineers  and 
customer  ratings  of  the  accepted  messages,  it  appears  that  such  channels 
as  vendors,  and  analysis  and  experimentation  are  over-utilized  relative 
to  their  effectiveness  and  information  from  company  research  programs 
and  experts  on  the  technical  staff  are  under-utilized. 

4.  Literature  is  not  greatly  used,  and  what  is  used  consists  primarily  of 
trade  publications  rather  than  professional  journals. 
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